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ABSTRACT 

This study's interests dictate which statements researchers 
attend to during interviews and which ones researchers consider to constitute 
data during later analysis . The paper shows that hearing all of what students 
are saying requires careful effort. Explicit consideration of possible 
research agendas is suggested to increase opportunities for productive 
research. (Author/KHR) 
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The challenge of listening: The effect of researcher agenda 
on data collection and interpretation 

Rachel E. Scherr, University of Maryland, College Park MD 20742^41 1 1, rescherr@physics.umd.edu 
Michael C. Wittmann, University of Maine, Orono ME 04469-5709, wittmann@umit.maine.edu 



Even in the relatively favorable circumstances of an individual interview, accurate listening 
requires careful effort. Our research interests dictate which student statements we attend to 
during the interview, and which we consider to constitute data during later analysis. Explicit 
consideration of possible research agendas can increase opportunity for productive research. 



Introduction 

As researchers and as instructors, we 
face the daily challenge of diagnosing 
student ideas by interpreting their written 
or spoken statements. Individual student 
interviews are often considered the gold 
standard for listening accurately to stu- 
dent ideas. However, even in an open- 
ended, time -unlimited, one-on-one con- 
versation, accurate listening requires care- 
ful effort. We can and do ignore student 
statements when our own research agenda 
limits our attention. Explicit considera- 
tion of possible research agendas can in- 
crease our awareness of the richness of 
interview data. 

The authors of this paper have multi- 
ple roles in what follows. One author 
(MCW) conducted the interview cited 
here as part of a larger investigation and 
has since, with collaborators, published a 
paper describing his findings . 1 The pri- 
mary author (RES) analyzed the interview 
transcript some time later. In this paper, 
MCW is sometimes referred to as “the 
interviewer,” and RES is sometimes re- 
ferred to as “the observer.” 

An Interview on Current and 
Conductivity 

In a previous paper , 1 MCW and his 
collaborators described a set of interviews 
in which students were given a simple 
apparatus (a battery with two leads) to 
which they could attach steel wire, copper 
wire, rubber bands, or wood. Students 
were asked to describe what happens in 
(e.g.) the steel wire when it is connected 
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to the battery terminals. 

Both the physics content and the find- 
ings of the interviews regarding student 
models of electrical conduction are de- 
scribed in detail in the cited paper. Our 
interest here is not in student conceptual 
understanding of current, but in the nego- 
tiation between the interviewer and the 
subject as they engage in the interview 
task. For this purpose we cite an excerpt 
from the beginning of a representative 
interview and pose the following ques- 
tions: How is this interview going so far? 
On what basis do you judge it to be going 
well or poorly? Does your answer depend 
on whether you are looking at physics 
content knowledge, interactions between 
interviewer and student, or the student’s 
views about knowledge in this topic area? 

Interviewer: So here is the set-up .. battery, 
your average battery pack, we’ve got two 
leads, and let's say that I took a piece of 
stainless steel, and I placed the stainless 
steel into the circuit; I made a circuit out 
of it, right? So attach one end, attach the 
other end ... what happens when you at- 
tach both ends? 

Student: The circuit loop is complete, so cur- 
rent or electrons will flow out and around 
back in. 

I: Okay. What's going on specifically inside 
of this piece right here? [indicates steel 
rod] 

S: Electrons are also flowing through that, 
depending on the makeup of this, I don't 
know, assuming just... steel, you said... 
conducting material. I don't know if that's 
what you... complete conductor, semi- 
conductor. Do you want me to explain in 
more detail...? 
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I: Yeah, sure... go into as much detail as you 
can. Feel free to make any type of draw- 
ings along the way that might help you in 
your explanation. 

S: Okay. I don't know exactly what happens 
in a resistor, but current is slowed down. 
It slows down... well, I would assume that 
there's some kind of chemical property, 
well I don't know if it's a chemical prop- 
erty, but the oscillating electrons, maybe 
they're not oscillating as much ... 

I: What do you mean by oscillating elec- 
trons? 

S: Well, that goes back to the first pretestand 
how the electrons move around the wire, I 
think the electrons are oscillating, slowly 
moving around. 

I: Okay, so. . .moving around the circuit? 

S: Yeah, that's right. Whatever. 

I: [...] How are they oscillating? Around 
what? Can you give me more detail? 

S: Mmm...not much! I wouldn't know if 
they were. . .well, oscillating perpendicular 
to the flow or parallel... I don't know, are 
they oscillating at all? 

[Interviewer asks what would be different if 
the rod were copper.] 

S: Well, all these things have uh, these are 
different chemicals, and they're each going 
to have their different electrons and pro- 
tons, so when the electron moves through, 
maybe there's the attraction and repulsion 
between this structure, where they're all 
positive and negative electrons ... that's 
stuff I've never thought about before, and 
I'm just making it up as we go along! 

I: Well, okay! See, that's not bad... after all 
there's ... it's how you come to making it 
up that's interesting also . 2 

The effect of observer agenda 
on data interpretation 

We pose again the questions of inter- 
est to us in this analysis: How is this in- 
terview going so far? On what basis do 
you judge it to be going well (or poorly)? 

The observer (RES) read the entire in- 
terview some time after the interview had 
occurred and initially found the excerpted 
portion easy to judge: it was not very in- 
teresting. What was lacking, in her initial 



opinion, was nformation about how the 
student thinks conduction works. “Sarah” 
(an alias) cffers little sense of a physical 
mechanism for current; the electrons just 
‘move around the loop.’ She refers to 
‘oscillation/ but does not say why any- 
thing is oscillating, or what the oscillation 
has to do with current. Worse, the inter- 
viewer seems to be having some difficulty 
getting her to discuss such details; he asks 
“what she means by oscillating electrons,” 
for example, but her response (“the elec- 
trons are oscillating, slowly moving 
around,”) is not clarifying. 

Other researchers may agree or dis- 
agree with the observer’s initial judgment; 
it does not necessarily tell us the real 
character of the interview excerpt. What 
the judgment does tell us is the nature of 
the observer ’s interests. Her automatic 
interest is, apparently, in student concep- 
tual understanding of physical mecha- 
nism. The observer’s interest is only one 
of a number of primary interests that ob- 
servers might have (or cultivate). Below 
we offer an incomplete list of possible 
research agendas. 

1. Conceptual knowledge of physical 
mechanism 

Does the student know the correct phys- 
ics? Does she have a particular alterna- 
tive model? 

2. Source of knowledge 

Is the student’s knowledge memorized, 
constructed, experienced with the senses? 

3. Knowledge construction 

Is the student skilled at it? How does she 
generate and select among ideas? 

4. Beliefs about knowledge 

What does her speech reveal about her 
epistemological stance? 

The observer apparently had item 1 
(“Conceptual knowledge of physical 
mechanism”) as her primary research 
agenda, and since there is relatively little 
information in the excerpt that is relevant 
to that interest, it was reasonable for her 
to judge the interview excerpt as lacking. 
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An observer with item 2 (“Source of 
knowledge”) as her research agenda, 
however, might judge the excerpt to be a 
rich source of information. For example, 
about halfway through the excerpted por- 
tion the student says, “Well, that goes 
back to the first pretest;” she’s drawing on 
issues raised earlier in her physics class. 
Later she claims, ‘We're trying to go to 
organic chemistry, you know, to draw all 
these funny pictures,” and sketches some- 
thing resembling a lattice structure. An 
observer interested in item 3, “Knowledge 
construction,” might be especially inter- 
ested in the following sequence somewhat 
later in the interview, in which Sarah ex- 
plores an apparently new idea: 

I: Okay. If I were to put in a different mate- 
rial... rather than steel, I put in copper. 

S: Different chemical makeup, so it could 
. . .the electrons could either go faster or 
slower.I: Okay, what part of the 

chemical makeup might actually deter- 
mine that? 

[...] 

S: Well, all these things have uh, these are 
different chemicals, and they're each go- 
ing to have their different electrons and 
protons, so when the electron moves 
through, maybe there's the attraction and 
repulsion between this structure, where 
they're all positive and negative elec- 
trons.An observer whose interests 
were primarily epistemological (item 4) 
would pay special gttention when Sarah 
says, “That’s stuff I’ve never thought 
about before, and I’m just making it up as 
I go along!” 

We have coded each of Sarah’s turns 
at talk to indicate what sort of information 
they contain: information about her con- 
ceptual knowledge of the physical 
mechanism, information about her 
sources of knowledge, and so on. 3 We 
found that within Sarah’s first eighteen 
turns at talk, Sarah made nineteen utter- 
ances that were codable according to the 
four categories described. Of those nine- 
teen utterances, six were relevant to 
describing a physical mechanism, five to 



scribing a physical mechanism, five to 
sources of knowledge, five to knowledge 
construction, and four to epistemological 
issues. To pay attention to only one of 
these four categories is to fail to hear 
much of what the student is saying. 

Our coding scheme is nontrivial to 
execute, and we are still examining issues 
of inter-rater reliability (our coding repre- 
sents consensus after discussion). Our 
intention, however, is not to claim that we 
have definitively identified the nature of 
each of Sarah’s statements. Instead, we 
intend only to illustrate that different re- 
search agendas can result in very different 
judgments of the interview eccerpt. In 
particular, an observer ’s interests dictate 
which student statements are considered 
to constitute data . The observer’s initial 
judgment - that the interview excerpt did 
not contain much hformation - reveals 
that she did not initially attend to agendas 
other than #1 (conceptual knowledge of 
physical mechanism). Her interests acted 
as a “filter” on the data. 

The effect of interviewer agenda 
on data collection 

The observation that an observer’s re- 
search agenda may affect data interpreta- 
tion naturally raises questions about the 
research agenda of the interviewer. It is 
reasonable to assume that the interviewer 
has a “filter” as well, and that his interpre- 
tation of student statements during the 
interview shapes the course of the conver- 
sation to some extent. Because research 
agendas are often implicit, it is rarely suf- 
ficient to simply ask the interviewer about 
his intentions. Instead, we examine the 
transcript for clues to his interests. 

We have coded the first eighteen in- 
terviewer turns at talk to indicate what 
sort of information the interviewer was 
requesting of the student. The great ma- 
jority of die codable prompts (12/14) are 
for conceptual knowledge of physical 
mechanism. For example, the interviewer 
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asks, “What do you mean by oscillating 
electrons? ...How are they cscillating? 
Around what?” There are two prompts 
for knowledge construction ( e.g “It’s 
how you come to making it up that’s in- 
teresting also”), but the interviewer’s pri- 
mary interest appears to be in the first of 
the four areas cited above. 

Additional evidence for this identifi- 
cation appears in the published paper de- 
scribing these interviews. In that article, 
the authors represent Sarah in the follow- 
ing way: 

“Sarah first described atomic lattice vibra- 
tions in a heated wire impeding electron 
flow, but then changed her response to say 
that the energy, when transferred to elec- 
trons, helped the electron flow.” 

The authors make other comments 
about Sarah that indicate they were also 
paying Mention to research agenda #3 
(knowledge construction). However, their 
primary interest appears to lie in Sarah’s 
conceptual model for electrical conduc- 
tion. 

The effect of the interviewer’s re- 
search agenda on data collection appears 
to be similar to the effect of the observer’s 
agenda on data interpretation: the inter- 
viewer shows selective attention to par- 
ticular student statements. Consider, for 
example, the following exchange: 

S: Okay. I don't know exactly what happens 
in a resistor, but current is slowed down. 
It slows down... well, I would assume that 
there's some kind of chemical property, 
well I don't know if it's a chemical prop- 
erty, but the oscillating electrons, maybe 
they're not oscillating as much ... 

I: What do you mean by oscillating elec- 
trons? 

Sarah’s statement includes both hints of a 
conceptual model (“oscillating electrons”) 
and information regarding the sources of 
her knowledge (“I don’t know exactly 
what happens... there’s some kind of a 
chemical property”). The interviewer 



might make any one of several possible 
moves at this point; for example, he might 
ask Sarah what she means by a chemical 
property, or whether it’s legitimate to 
speculate about mechanisms one doesn’t 
remember exactly. Instead, though, he 
asks for more detail about the oscillations. 
His choice of response calls our attention 
to his primary interest in the conceptual 
content of Sarah’s ideas. And, presuma- 
bly, his response calls th z student *s atten- 
tion to his interests also, as she and he 
together negotiate the course of the inter- 
view. His choice of response encourages 
her to provide more conceptual informa- 
tion in the future. 

Discussion 

We hope to have shown in this paper 
that hearing all of what students are say- 
ing requires careful effort. We are not 
automatically conscious of everything a 
student says; the “filter” of our own re- 
search interests blocks some student 
statements. Such a filter is not inappro- 
priate for researchers; specialization is 
usually necessary for cfetailed analysis. 
However, to the extent that our research 
agendas are unexamined, they may con- 
trol our attention inappropriately. Con- 
scious consideration of possible research 
agendas widens the range of opportunity 
for productive research. 
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